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congressional  requesters 


Why  GAO  Did  This  Study 

DOD  officials  have  stated  that  the 
implementation  of  enterprise  resource 
planning  systems,  such  as  GCSS- 
Army,  is  critical  to  the  department’s 
goal  of  correcting  financial 
management  deficiencies  and  ensuring 
that  its  financial  statements  are 
validated  as  audit  ready  by 
September  30,  2017,  as  called  for  by 
the  National  Defense  Authorization  Act 
for  Fiscal  Year  2010. 

GAO  was  asked  to  review  the 
schedule  and  cost  estimates  for 
selected  DOD  systems.  This  report 
addresses  the  extent  to  which  the 
schedule  and  cost  estimates  for 
GCSS-Army  were  prepared  consistent 
with  GAO’s  Schedule  and  Cost 
Guides.  The  schedule  and  cost 
estimates  are  designed  to  cover 
GCSS-Army  implementation  through 
2017. 

GAO  assessed  the  schedule  and  cost 
estimates  that  supported  DOD’s 
December  2012  full  deployment 
decision,  which  granted  approval  for 
GCSS-Army  to  be  deployed  for 
operational  use  to  all  remaining 
locations.  GAO  also  met  with  GCSS- 
Army  program  officials,  including  lead 
schedulers  and  cost  estimators. 

What  GAO  Recommends 

GAO  is  making  two  recommendations 
aimed  at  improving  the  Army’s 
implementation  of  schedule  and  cost 
best  practices  for  GCSS-Army.  DOD 
concurred,  but  the  completed  actions  it 
described  related  to  the  cost  estimate 
were  not  fully  responsive  to  GAO’s 
recommendation.  GAO  continues  to 
believe  that  fully  incorporating  best 
practices  in  the  cost  estimate  would 
help  improve  its  reliability. 

View  GAO-14-470.  For  more  information, 
contact  Asif  A.  Khan  at  (202)  512-9869  or 
khana@gao.gov  or  Nabajyoti  Barkakati  at 
(202)  512-4499  or  barkakatin@gao.gov. 


DOD  BUSINESS  SYSTEMS  MODERNIZATION 

Additional  Enhancements  Are  Needed  for  Army 
Business  System  Schedule  and  Cost  Estimates  to 
Fully  Meet  Best  Practices 


What  GAO  Found 

The  Army  made  some  improvements  to  its  schedule  and  cost  estimates  that 
supported  the  December  2012  full  deployment  decision  for  the  Global  Combat 
Support  System-Army  (GCSS-Army);  however,  the  schedule  and  cost  estimates 
did  not  fully  meet  best  practices.  GAO  found  that  the  schedule  substantially  met 
the  credibility  and  control  characteristics  for  developing  a  high-quality  and  reliable 
schedule.  For  example,  the  schedule  was  horizontally  integrated,  which  means 
that  it  links  products  and  outcomes  with  other  associated  sequenced  activities.  In 
addition,  the  GCSS-Army  program  management  officials  followed  general 
guidelines  for  updating  the  schedule  on  a  regular  basis.  GAO  found  that  the 
schedule  partially  met  the  comprehensiveness  and  construction  characteristics 
for  a  reliable  schedule.  Specifically,  resources  were  not  assigned  to  specific 
activities,  and  the  schedule  lacked  a  valid  critical  path,  preventing  management 
from  focusing  on  the  activities  most  likely  to  have  detrimental  effects  on  key 
program  milestones  if  not  completed  as  planned.  By  incorporating  best  practices 
for  developing  a  reliable  schedule,  the  Department  of  Defense  (DOD)  would 
increase  the  probability  of  completing  the  GCSS-Army  program  by  the  projected 
date. 


Extent  to  Which  Global  Combat  Support  System-Army  Schedule  Met  Best  Practices 

Characteristic 

Assessment 

Comprehensive 

Partially  met 

Well-constructed 

Partially  met 

Credible 

Substantially  met 

Controlled 

Substantially  met 

Source:  GAO  analysis  based  on  information  provided  by  the  Army.  |  GAO-14-470 


GAO  found  that  the  GCSS-Army  cost  estimate  fully  or  substantially  met  the 
comprehensiveness,  documentation,  and  accuracy  characteristics  of  a  high- 
quality  and  reliable  cost  estimate.  For  example,  the  cost  estimate  included  both 
government  and  contractor  costs  for  the  program  over  its  life  cycle,  provided 
documentation  that  substantially  described  detailed  calculations  used  to  derive 
each  element’s  cost,  and  was  adjusted  for  inflation.  In  addition,  GAO  found  that 
the  cost  estimate  partially  met  the  credibility  characteristic  of  a  reliable  cost 
estimate.  Although  program  management  officials  provided  a  cost  model  that 
discussed  a  limited  risk  analysis,  the  results  of  the  risk  and  uncertainty  analysis 
were  not  documented.  Incorporating  best  practices  would  help  ensure  that  DOD 
has  a  reliable  cost  estimate  that  provides  the  basis  for  effective  resource 
allocation,  proactive  course  correction  when  warranted,  and  accountability  for 
results. 


Extent  to  Which  Global  Combat  Support  System-Army  Cost  Estimate  Met  Best  Practices 

Characteristic 

Assessment 

Comprehensive 

Fully  met 

Well-documented 

Substantially  met 

Accurate 

Substantially  met 

Credible 

Partially  met 

Source:  GAO  analysis  based  on  information  provided  by  the  Army.  |  GAO-14-470 
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Abbreviations 


DASA-CE 

DOD 

ERP 

GCSS-Army 

IMS 


Deputy  Assistant  Secretary  of  the  Army  for  Cost 
and  Economics 
Department  of  Defense 
enterprise  resource  planning 
Global  Combat  Support  System-Army 
integrated  master  schedule 
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GAO 

441  G  St.  N.W. 
Washington,  DC  20548 


U.S.  GOVERNMENT  ACCOUNTABILITY  OFFICE 


September  30,  2014 
Congressional  Requesters 

The  Department  of  Defense  (DOD)  invests  billions  of  dollars  annually  to 
develop  and  implement  enterprise  resource  planning  (ERP)  systems, 
which  it  considers  critical  to  transforming  the  department’s  business 
operations  and  addressing  some  of  its  long-standing  weaknesses, 
including  those  related  to  financial  management  and  business  systems 
modernization.1  DOD  officials  have  stated  that  the  implementation  of  the 
ERPs,  such  as  the  Global  Combat  Support  System-Army  (GCSS-Army), 
is  a  key  component  to  the  department’s  goal  of  correcting  financial 
management  deficiencies  and  ensuring  that  its  financial  statements  are 
validated  as  audit  ready  by  September  30,  201 7, 2  as  called  for  by  the 
National  Defense  Authorization  Act  for  Fiscal  Year  2010. 3  DOD’s 
business  systems4  modernization  program  has  been  on  GAO’s  high-risk 
list  since  1995  because  of  the  size,  complexity,  and  significance  of  the 
related  efforts.5 

In  October  2010,  we  reported  that  the  Army  did  not  fully  follow  best 
practices  in  developing  a  reliable  schedule  or  cost  estimate  for  GCSS- 
Army.6  Specifically,  the  Army  had  not  developed  a  fully  integrated  master 


ERP  system  is  an  automated  system  using  commercial  off-the-shelf  software 
consisting  of  multiple,  integrated  functional  modules  that  perform  a  variety  of  business- 
related  tasks,  such  as  general  ledger  accounting,  payroll,  and  supply  chain  management. 

2For  a  complete  list  of  the  ERP  systems  that  DOD  has  identified  as  essential  to 
transforming  its  business  operations,  see  GAO,  DOD  Financial  Management:  Reported 
Status  of  Department  of  Defense’s  Enterprise  Resource  Planning  Systems ,  GAO-1 2-565R 
(Washington,  D.C.:  Mar.  30,  2012). 

3Pub.  L.  No.  111-84,  div.  A,  §  1003(a),  (b)  (Oct.  28,  2009).  Further,  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2014,  Pub.  L.  No.  113-66,  div.  A,  §  1003  (Dec.  26,  2013), 
also  mandates  a  full  audit  of  DOD’s  fiscal  year  2018  financial  statements,  and  that  those 
results  be  submitted  to  Congress  by  March  31, 2019. 

4DOD’s  business  systems  are  information  systems,  including  financial  and  nonfinancial 
systems  that  support  DOD  business  operations,  such  as  civilian  personnel,  finance, 
health,  logistics,  military  personnel,  procurement,  and  transportation. 

5GAO,  High-Risk  Series:  An  Update,  GAO-13-283  (Washington,  D.C.:  February  2013). 

6GAO,  DOD  Business  Transformation:  Improved  Management  Oversight  of  Business 
System  Modernization  Efforts  Needed,  GAO-1 1-53  (Washington,  D.C.:  Oct.  7,  2010). 
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schedule  that  reflected  all  activities,  including  government  and  contractor 
activities.  Having  such  a  schedule  is  crucial  to  the  Army’s  ability  to  reliably 
estimate  the  program  completion  date.  Also,  the  Army  did  not  perform  a 
sensitivity  analysis  for  the  cost  estimate,  which  would  have  helped 
decision  makers  in  determining  how  changes  to  assumptions  or  key  cost 
drivers  could  affect  the  credibility  of  the  cost  estimate. 

To  support  Congress’s  continuing  oversight  of  DOD’s  progress  in 
implementing  its  ERP  systems,  you  asked  us  to  review  the  schedule  and 
cost  estimates  for  selected  DOD  ERP  systems.  The  objective  of  this 
review  was  to  determine  the  extent  to  which  the  schedule  and  cost 
estimates  for  GCSS-Army  were  prepared  consistent  with  GAO’s 
Schedule  and  Cost  Guides.7  We  reviewed  the  most  current  GCSS-Army 
schedule  and  cost  estimates  available  at  the  time  of  our  review,  which 
supported  DOD’s  December  2012  full  deployment  decision.8 


Scope  and 
Methodology 


We  assessed  the  GCSS-Army  schedule9  that  supported  DOD’s 
December  2012  full  deployment  decision  using  the  GAO  Schedule  Guide 
to  determine  whether  it  was  comprehensive,  well-constructed,  credible, 
and  controlled.10  To  assess  the  schedule,  we  obtained  and  reviewed 
documentation,  including  the  integrated  master  plan,  work  breakdown 


7 GAO,  GAO  Schedule  Assessment  Guide:  Best  Practices  for  Project  Schedules — 
Exposure  Draft,  GAO-12-120G  (Washington,  D.C.:  May  2012),  and  GAO  Cost  Estimating 
and  Assessment  Guide:  Best  Practices  for  Developing  and  Managing  Capital  Program 
Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 

SA  full  deployment  decision  is  the  decision,  following  completion  of  operational  testing  of 
the  program,  to  scale  up  production  and/or  fielding. 

9Although  the  Army  provided  us  with  its  integrated  master  schedule  (IMS)  for  the  GCSS- 
Army  program,  we  determined  that  it  did  not  meet  all  of  the  characteristics  of  an  IMS  in 
accordance  with  GAO’s  Schedule  Guide.  Specifically,  an  IMS  is  a  document  that 
integrates  the  planned  work  by  the  government,  contractor,  and  other  key  parties;  the 
resources  necessary  to  accomplish  that  work;  and  the  associated  budget.  For  purposes  of 
this  report,  we  use  “schedule”  to  refer  to  the  Army’s  IMS. 

1 0 

A  schedule  is  controlled  if  it  is  updated  periodically  by  trained  schedulers  to  realistically 
forecast  dates  for  program  activities  and  compared  against  a  designated  baseline 
schedule  to  measure,  monitor,  and  report  the  project’s  progress. 
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structure,  and  statement  of  work.11  To  assess  the  program’s  cost 
estimate,  we  used  the  GAO  Cost  Guide  to  evaluate  the  GCSS-Army 
Program  Management  Office’s  estimating  methodologies,  assumptions, 
and  results  to  determine  whether  the  cost  estimate  was  comprehensive, 
well-documented,  accurate,  and  credible.  We  obtained  and  reviewed 
documentation,  including  the  program  office  estimate,  software  cost 
model,  independent  cost  estimate,  and  risk  and  uncertainty  analysis.12 
We  also  met  with  key  program  officials,  such  as  the  program  manager, 
lead  schedulers,  and  cost  estimators  to  present  the  preliminary  results  of 
our  assessment  of  the  program’s  schedule  and  cost  estimates  best 
practices  and  obtained  explanations  and  clarifications. 

We  conducted  this  performance  audit  from  October  201 1  to  September 
2014  in  accordance  with  generally  accepted  government  auditing 
standards.13  Those  standards  require  that  we  plan  and  perform  the  audit 
to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis 
for  our  findings  and  conclusions  based  on  our  audit  objectives.  We 
believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives. 


^An  integrated  master  plan  provides  an  event-based  hierarchy  of  program  events,  with 
each  event  supported  by  accomplishments  and  each  accomplishment  associated  with 
specific  criteria  to  be  satisfied  for  its  completion.  The  plan  is  normally  part  of  the  contract 
and  is  therefore  contractually  binding.  A  program’s  work  breakdown  structure  defines  in 
detail  the  work  necessary  to  accomplish  a  program’s  objectives,  including  activities  to  be 
performed  by  both  the  government  and  contractors.  A  statement  of  work  accounts  for  the 
detailed  technical  tasks  and,  when  completed,  facilitates  communication  between  the 
customer  and  supplier  on  cost,  schedule,  technical  information,  and  the  progress  of  the 
work. 

12An  independent  cost  estimate  is  another  estimate  based  on  the  same  technical 
information  that  is  used  to  validate  and  cross-check  the  cost  estimate,  but  is  prepared  by  a 
person  or  organization  that  has  no  stake  in  the  approval  of  the  program.  A  risk  and 
uncertainty  analysis  recognizes  the  potential  for  error  and  attempts  to  quantify  it  by 
identifying  the  effects  of  changing  key  cost  drivers  (e.g.,  costs  associated  with  personnel 
or  software  development). 

When  we  initiated  this  work  in  October  2011,  the  U.S.  Army  Program  Management 
Office  was  unable  to  provide  us  with  a  current  GCSS-Army  cost  estimate  or  schedule  for 
our  review.  We  therefore  suspended  the  engagement  from  May  2012  to  February  2013  to 
give  the  U.S.  Army  Program  Management  Office  more  time  to  produce  the  GCSS-Army 
cost  estimate  and  schedule.  The  Program  Management  Office  provided  us  with  these  in 
February  2013. 
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Background 


GCSS-Army  was  initiated  in  December  2003  and  is  intended  to  provide 
all  active  Army,  National  Guard,  and  Army  Reserve  tactical  units  with  the 
capability  to  track  supplies,  spare  parts,  and  organizational  equipment. 
The  system  is  also  intended  to  track  unit  maintenance,  total  cost  of 
ownership,  and  other  financial  transactions  related  to  logistics  for  all  Army 
units — about  160,000  users.  GCSS-Army  is  intended  to  integrate 
approximately  40,000  local  supply  and  logistics  databases  into  a  single, 
enterprise-wide  system.  In  December  2012,  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics  granted  full 
deployment  decision  approval  for  GCSS-Army  to  be  deployed  to  all 
remaining  locations  beyond  the  limited  fielding  locations  of  the  life-cycle 
acquisition  process.14  DOD  officials  reported  that  the  GCSS-Army  full 
deployment  will  be  completed  by  the  fourth  quarter  of  fiscal  year  201 7. 
GCSS-Army  program  functionality  is  intended  to  be  implemented  across 
the  Army  in  two  waves — the  first  is  to  include  two  releases  and  is  to 
provide  supply  (warehouse)  and  financial  reporting  capabilities,  and  the 
second  is  to  include  one  release,  which  is  to  provide  property  book15  and 
maintenance  capabilities.  DOD  has  approved  the  funding  for  the  Army  to 
proceed  with  the  deployment  of  the  GCSS-Army  functionality  to  all 
intended  locations.  This  funding  is  approximately  $3.7  billion.  The  Army 
reported  that  it  had  spent  about  $1.6  billion  as  of  June  30,  2014. 

In  October  2010,  we  reported  that  the  Army  did  not  fully  follow  best 
practices  in  developing  a  reliable  schedule  and  cost  estimate  for 
implementing  GCSS-Army.16  In  particular,  the  Army  had  not  developed  a 
fully  integrated  master  schedule  that  reflected  all  government  and 
contractor  activities  and  had  not  performed  a  sensitivity  analysis  for  the 
cost  estimate.  We  recommended  that  the  Army  develop  an  integrated 
master  schedule  that  fully  incorporated  best  practices,  such  as  capturing 
all  activities,  sequencing  all  activities,  integrating  activities  horizontally 
and  vertically,17  establishing  the  critical  path  for  all  activities,  and 


14Limited  fielding  is  the  deployment  of  a  capability  to  a  limited  number  of  users  to  test  the 
capability  in  an  operational  environment. 

15Property  book  is  a  formally  designated  set  of  property  records  used  to  account  for 
organizational  and  installation  property  in  a  using  unit. 

16GAO-1 1-53. 

17A  horizontally  integrated  schedule  links  products  and  outcomes  with  other  associated 
sequenced  activities,  and  a  vertically  integrated  schedule  ensures  that  the  start  and 
completion  dates  for  activities  are  aligned  with  such  dates  on  subsidiary  schedules 
supporting  tasks  and  subtasks. 
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conducting  a  schedule  risk  analysis.  In  addition,  we  recommended  that 
the  Army  update  the  cost  estimate  by  using  actual  costs  and  preparing  a 
sensitivity  analysis.  DOD  concurred  with  our  recommendations,  and  this 
report  provides  the  status  of  the  department’s  efforts  to  address  our  prior 
recommendations. 


GAO’S  Cost  and  Schedule  In  March  2009,  we  published  the  Cost  Guide  to  address  a  gap  in  federal 
Guides  guidance  about  processes,  procedures,  and  practices  needed  to  ensure 

reliable  cost  estimates.18  The  Cost  Guide  provides  a  consistent 
methodology  based  on  best  practices  that  can  be  used  across  the  federal 
government  to  develop,  manage,  and  evaluate  capital  program  cost 
estimates.  The  methodology  is  a  compilation  of  characteristics  and 
associated  best  practices  that  federal  cost  estimating  organizations  and 
industry  use  to  develop  and  maintain  reliable  cost  estimates  throughout 
the  life  of  an  acquisition  program. 

In  May  2012,  we  issued  an  exposure  draft  of  the  Schedule  Guide  as  a 
companion  to  the  Cost  Guide.19  A  consistent  methodology  for  developing, 
managing,  and  evaluating  capital  program  cost  estimates  includes  the 
concept  of  scheduling  the  necessary  work  to  a  timeline,  as  discussed  in 
the  Cost  Guide.  Simply  put,  schedule  variances  are  usually  followed  by 
cost  variances.  Because  some  program  costs,  such  as  labor,  supervision, 
rented  equipment,  and  facilities,  cost  more  if  the  program  takes  longer,  a 
reliable  schedule  can  contribute  to  an  understanding  of  the  cost  impact  if 
the  program  does  not  finish  on  time.  In  addition,  management  tends  to 
respond  to  schedule  delays  by  adding  more  resources  or  authorizing 
overtime.  Further,  a  schedule  risk  analysis  allows  for  program 
management  to  account  for  the  cost  effects  of  schedule  slippage  when 
developing  the  life-cycle  cost  estimate.  A  cost  estimate  cannot  be 
considered  fully  credible  if  it  does  not  account  for  the  cost  effects  of 
schedule  slippage. 


18GAO-09-3SP. 

19GAO-12-120G. 
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GCSS-Army  Program 
Schedule  and  Cost 
Estimates  Did  Not 
Fully  Meet  Best 
Practices 


We  found  that  the  program  schedule  and  cost  estimates  for  the  GCSS- 
Army  did  not  fully  meet  best  practices.  Specifically,  the  GCSS-Army 
schedule  supporting  the  December  2012  full  deployment  decision  partially 
met  the  comprehensiveness  and  construction  characteristics  and 
substantially  met  the  credibility  and  control  characteristics  for  developing 
a  high-quality  and  reliable  schedule.  In  addition,  the  cost  estimate  fully 
met  the  comprehensiveness  characteristic,  substantially  met  the 
documentation  and  accuracy  characteristics,  and  partially  met  the 
credibility  characteristic  for  developing  a  high-quality  and  reliable  cost 
estimate.  It  is  important  that  the  schedule  and  cost  estimates  are 
continually  updated  throughout  the  program’s  life  cycle  so  that 
management  has  the  best  information  available  to  make  decisions.  By 
incorporating  best  practices  for  developing  reliable  schedule  and  cost 
estimates,  DOD  would  increase  the  probability  of  GCSS-Army 
successfully  achieving  full  deployment  by  the  fourth  quarter  of  fiscal  year 
2017  to  provide  needed  functionality  for  financial  improvement  and  audit 
readiness. 


Program  Schedule  Did  Not  Our  analysis  found  that  the  GCSS-Army  program  substantially  met  two 
Fully  Meet  Best  Practices  and  partially  met  the  other  two  characteristics  of  a  reliable  schedule 

estimate  and  therefore  did  not  provide  the  information  needed  to  support 
the  December  2012  full  deployment  decision  (see  table  1).  Appendix  I 
contains  our  detailed  analysis  of  the  GCSS-Army  schedule  estimate.  The 
success  of  any  program  depends  on  having  a  reliable  schedule  of  the 
program’s  work  activities  that  will  occur,  how  long  they  will  take,  and  how 
the  activities  are  related  to  one  another.  As  such,  the  schedule  not  only 
provides  a  road  map  for  systematic  execution  of  a  program,  but  also 
provides  the  means  by  which  to  gauge  progress,  identify  and  address 
potential  problems,  and  promote  accountability. 
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Table  1 :  Extent  to  Which  Global  Combat  Support  System-Army  Schedule  Met  Best 
Practices 

Characteristic 

Assessment 

Comprehensive 

Partially  met 

Well-constructed 

Partially  met 

Credible 

Substantially  met 

Controlled 

Substantially  met 

Source:  GAO  analysis  based  on  information  provided  by  the  Army.  |  GAO-14-470 


Note:  GAO's  methodology  includes  five  levels  of  compliance  with  its  best  practices.  “Not  met”  means 
the  program  provided  no  evidence  that  satisfies  any  of  the  criterion.  “Minimally  met”  means  the 
program  provided  evidence  that  satisfies  a  small  portion  of  the  criterion.  “Partially  met”  means  the 
program  provided  evidence  that  satisfies  about  half  of  the  criterion.  “Substantially  met”  means  the 
program  provided  evidence  that  satisfies  a  large  portion  of  the  criterion.  “Fully  met”  means  the 
program  provided  evidence  that  completely  satisfies  the  criterion. 


Comprehensive.  A  schedule  should  reflect  all  activities  as  defined  in  the 
program’s  work  breakdown  structure,  including  activities  to  be  performed 
by  the  government  and  the  contractor;  the  resources  (e.g.,  labor, 
materials,  and  overhead)  needed  to  complete  each  activity;  and  how  long 
each  activity  will  take. 

We  found  that  the  GCSS-Army  schedule  partially  met  the  comprehensive 
characteristic.  The  schedule  used  to  support  the  full  deployment  decision 
reflected  all  activities  to  be  performed  by  both  the  government  and 
contractor  for  the  program.  However,  resources  were  not  loaded  into  the 
schedule  software  and  were  not  assigned  to  specific  activities  in  the 
schedule.  GCSS-Army  program  management  officials  told  us  that  the 
contractor  used  a  separate  system  outside  the  schedule  to  manage  the 
resources  needed  for  the  program.  Information  on  resource  needs  and 
availability  in  each  work  period  assists  the  program  office  in  forecasting 
the  likelihood  that  activities  will  be  completed  as  scheduled.  If  the  current 
schedule  does  not  allow  insight  into  the  current  or  projected  allocation  of 
resources,  the  risk  of  the  program’s  schedule  slipping  is  significantly 
increased.  Our  analysis  also  determined  that  activity  durations  were  not 
manageable  and  reasonably  estimated  in  the  schedule.  We  found  that  30 
percent  of  the  remaining  activities  in  the  schedule  exceeded  the  standard 
best  practice  for  activity  duration,  which  should  be  shorter  than 
approximately  44  working  days,  or  2  working  months.  For  example,  audit 
support  activities  had  durations  over  100  working  days.  Durations  should 
be  as  short  as  possible  to  facilitate  the  objective  measurement  of 
accomplished  effort.  If  activities  are  too  long,  the  schedule  may  not  have 
enough  detail  for  effective  progress  measurement  and  reporting. 
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Well-constructed.  A  schedule  should  be  planned  so  that  critical  project 
dates  can  be  met.  To  meet  this  objective,  all  activities  should  be  logically 
sequenced — that  is,  listed  in  the  order  in  which  they  are  to  be  carried  out. 
In  particular,  activities  that  must  finish  prior  to  the  start  of  other  activities 
(i.e. ,  predecessor  activities),  as  well  as  activities  that  cannot  begin  until 
other  activities  are  completed  (i.e.,  successor  activities),  should  be 
identified  and  their  relationships  established.  The  schedule  should  identify 
the  project’s  critical  path.20  Establishing  a  valid  critical  path  is  necessary 
for  examining  the  effects  of  any  activity  slipping  along  this  path.  The 
calculation  of  a  critical  path  determines  which  activities  drive  the  project’s 
earliest  completion  date.  The  schedule  should  also  identify  total  float  so 
that  the  schedule’s  flexibility  can  be  accurately  determined.21 

We  found  that  the  GCSS-Army  schedule  was  partially  well-constructed. 
The  majority  of  logic  used  to  sequence  the  activities  within  the  schedule 
was  generally  error  free,  clearly  indicating  to  program  management  the 
order  of  activities  that  must  be  accomplished.  However,  the  schedule’s 
critical  path  was  not  valid  because  it  included  level  of  effort  activities  and 
date  constraints.22  Level  of  effort  activities,  such  as  program 
management,  should  not  define  the  critical  path  because  they  are 
nondiscrete  support  activities  that  do  not  produce  a  definite  end  product; 
therefore,  level  of  effort  activities  cannot  determine  the  length  of  the 
project.  In  addition,  date  constraints  prevent  the  critical  path  from  being  a 
continuous  sequence  of  events  from  the  current  to  finish  dates  of  the 
project.  Rather  than  relying  on  such  constraints,  the  schedule  should  use 
logic  and  durations  in  order  to  reflect  realistic  start  and  completion  dates 
for  activities.  Successfully  identifying  the  critical  path  relies  on  several 
factors,  such  as  capturing  all  activities;  properly  sequencing  activities;  and 
assigning  resources,  which,  as  noted  earlier,  had  not  been  completely 
done.  Without  a  valid  critical  path,  management  cannot  focus  on  activities 
that  will  have  detrimental  effects  on  the  key  project  milestones  and 
deliverables  if  they  slip.  Further,  our  analysis  found  that  28  percent  of 


20A  critical  path  is  the  sequence  of  activities  that  represents  the  longest  path  from  the 
project’s  start  and  finish  dates. 

21  Float  is  the  amount  of  time  by  which  a  predecessor  activity  can  slip  before  the  delay 
affects  the  program’s  estimated  finish  date. 

22Level  of  effort  activities  are  typically  related  to  management  and  other  oversight  that 
continue  until  the  detailed  work  activities  they  support  have  been  completed.  A  constraint 
predefines  the  start,  finish,  or  both  dates  of  an  activity. 
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remaining  schedule  activities  had  more  than  100  working  days  of  total 
float,  meaning  that  those  activities  could  slip  almost  5  working  months 
and  not  affect  the  estimated  finish  date  of  the  program.  Based  on  the 
remaining  duration  of  the  program,  100  working  days  of  float  would  not 
appear  to  be  reasonable.  The  GCSS-Army  Program  Management  Office 
stated  that  total  float  was  not  reliable  at  the  time  of  the  full  deployment 
decision  because  the  schedule  was  being  updated  to  reflect  a 
modification  to  the  system.  Without  accurate  values  of  total  float  for  a 
program  activity,  management  cannot  determine  the  flexibility  of  tasks 
and  therefore  cannot  properly  reallocate  resources  from  tasks  that  can 
safely  slip  to  tasks  that  cannot  slip  without  adversely  affecting  the 
estimated  program  completion  date. 

Credible.  A  schedule  should  be  horizontally  and  vertically  integrated.  A 
horizontally  integrated  schedule  links  products  and  outcomes  with  other 
associated  sequenced  activities,  which  helps  verify  that  activities  are 
arranged  in  the  right  order  to  achieve  aggregated  products  or  outcomes. 
A  vertically  integrated  schedule  ensures  that  the  start  and  completion 
dates  for  activities  are  aligned  with  such  dates  on  subsidiary  schedules 
supporting  tasks  and  subtasks.  A  schedule  risk  analysis  should  also  be 
performed  using  statistical  techniques  to  predict  the  level  of  confidence  in 
meeting  a  program’s  completion  date. 

We  found  that  the  GCSS-Army  schedule  was  substantially  credible.  The 
schedule  was  substantially  horizontally  integrated,  which  means  that 
outcomes  were  aligned  with  sequenced  activities.  The  schedule  was  also 
substantially  vertically  integrated;  we  were  able  to  trace  varying  levels  of 
activities  and  supporting  subactivities.  Such  mapping  or  alignment  among 
subsidiary  schedules  enables  different  groups — such  as  government 
teams  and  contractors — to  work  to  the  same  master  schedule,  and 
provides  assurance  that  the  representation  of  the  schedule  to  different 
audiences  is  consistent  and  accurate.  However,  our  analysis  found  that  a 
schedule  risk  analysis  had  not  been  fully  conducted.  GCSS-Army 
program  management  officials  provided  documentation  for  a  schedule 
risk  analysis,  but  we  noted  that  risk  analyses  were  not  performed  for  all 
supporting  activities  because  program  management  officials  stated  that 
the  program  fielding  schedule  was  not  finalized  at  the  time  of  the  full 
deployment  decision.  If  a  schedule  risk  analysis  is  not  conducted, 
program  management  cannot  determine  (1)  the  likelihood  that  the  project 
completion  date  will  occur,  (2)  how  much  schedule  risk  contingency  is 
needed  to  provide  an  acceptable  level  of  certainty  for  completion  by  a 
specific  date,  (3)  risks  most  likely  to  delay  the  project,  (4)  how  much 
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contingency  reserve  each  risk  requires,  and  (5)  the  activities  that  are 
most  likely  to  delay  the  project.23 

Controlled.  A  schedule  should  be  continually  updated  using  logic, 
durations,  and  actual  progress  to  realistically  forecast  dates  for  program 
activities.  A  schedule  narrative  should  accompany  the  updated  schedule 
to  provide  decision  makers  and  auditors  a  log  of  changes  and  their  effect, 
if  any,  on  the  schedule  time  frame.  The  schedule  should  be  analyzed 
continually  for  variances  to  determine  when  forecasted  completion  dates 
differ  from  planned  dates.  This  analysis  is  especially  important  for  those 
variations  that  affect  activities  identified  as  being  in  a  program’s  critical 
path  and  that  can  affect  a  scheduled  completion  date.  A  baseline 
schedule  should  be  used  to  manage  the  program  scope,  the  time  period 
for  accomplishing  it,  and  the  required  resources. 

We  found  that  the  GCSS-Army  schedule  was  substantially  controlled. 
GCSS-Army  program  management  officials  stated  that  they  met  weekly 
to  discuss  proposed  schedule  changes  and  update  the  schedule’s 
progress,  and  management  also  prepared  a  schedule  narrative  document 
that  contained  a  list  of  custom  fields  and  assumptions.  In  addition,  we 
found  no  anomalies  throughout  the  schedule  (e.g.,  activities  with  planned 
start  dates  scheduled  to  occur  in  the  past  and  activities  with  actual  finish 
dates  scheduled  to  occur  in  the  future).  However,  we  found  that  there  was 
not  a  documented  baseline  schedule  to  measure  program  performance 
against,  which  would  allow  management  to  monitor  any  schedule 
variances  that  affect  the  completion  of  work.  Without  a  formally 
established  baseline  schedule  to  measure  performance  against, 
management  cannot  identify  or  mitigate  the  effect  of  unfavorable 
performance. 

In  our  October  2010  report,  we  recommended  that  the  Army  develop  an 
integrated  master  schedule  that  fully  incorporated  best  practices,  such  as 
capturing  all  activities,  sequencing  all  activities,  integrating  activities 
horizontally  and  vertically,  establishing  the  critical  path  for  all  activities, 


oo 

Contingency  represents  time  reserve  held  at  or  above  the  government  program 
management  office  for  “unknown  unknowns”  that  are  outside  a  contractor’s  control.  In  this 
context,  schedule  contingency  is  added  to  a  schedule  to  allow  for  items,  conditions,  or 
events  for  which  the  state,  occurrence,  or  effect  is  uncertain  and  for  which  experience 
implies  likely  delays.  Schedule  reserve,  in  contrast,  is  for  “known  unknowns”  that  are  tied 
to  the  contract’s  scope  and  managed  at  the  contractor  level. 
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and  conducting  a  schedule  risk  analysis.24  The  Army’s  December  2012 
GCSS-Army  schedule  used  to  support  the  full  deployment  decision 
addressed  several  of  the  best  practices  that  were  an  issue  in  our  prior 
report,  including  capturing  all  activities,  sequencing  all  activities,  and 
integrating  activities  horizontally  and  vertically.  However,  as  discussed, 
we  continued  to  identify  several  best  practices  that  were  not  yet  fully 
addressed  and  also  identified  several  new  areas  where  the  2012 
schedule  did  not  incorporate  best  practices,  such  as  activity  durations  and 
baseline  schedule.  Although  GCSS-Army  is  in  full  deployment,  without 
fully  addressing  best  practices  for  scheduling,  program  managers  will  not 
have  the  best  information  available  to  make  decisions  related  to  issues 
such  as  the  sequencing  of  activities  and  the  flexibility  of  the  schedule 
according  to  available  resources. 


Program  Cost  Estimate 
Did  Not  Fully  Meet  Best 
Practices 


We  found  that  the  GCSS-Army  program  fully  met  one,  substantially  met 
two,  and  partially  met  one  of  the  characteristics  of  a  reliable  cost  estimate 
and  therefore  did  not  provide  the  information  needed  to  support  the  full 
deployment  decision,  as  shown  in  table  2.  Appendix  II  contains  our 
detailed  analysis  of  the  GCSS-Army  cost  estimate.  A  reliable  cost 
estimate  is  critical  to  the  success  of  any  program  and  is  updated 
continually  throughout  its  life  cycle.  Such  an  estimate  provides  the  basis 
for  informed  investment  decision  making,  realistic  budget  formulation  and 
program  resourcing,  meaningful  progress  measurement,  proactive  course 
correction  when  warranted,  and  accountability  for  results. 


24GAO-1 1-53. 


Page  11 


GAO-14-470  DOD  Business  Systems  Modernization 


Table  2:  Extent  to  Which  Global  Combat  Support  System-Army  Cost  Estimate  Met 
Best  Practices 

Characteristic 

Assessment 

Comprehensive 

Fully  met 

Well-documented 

Substantially  met 

Accurate 

Substantially  met 

Credible 

Partially  met 

Source:  GAO  analysis  based  on  information  provided  by  the  Army.  |  GAO-14-470 


Note:  GAO's  methodology  includes  five  levels  of  compliance  with  its  best  practices.  “Not  met”  means 
the  program  provided  no  evidence  that  satisfies  any  of  the  criterion.  “Minimally  met”  means  the 
program  provided  evidence  that  satisfies  a  small  portion  of  the  criterion.  “Partially  met”  means  the 
program  provided  evidence  that  satisfies  about  half  of  the  criterion.  “Substantially  met”  means  the 
program  provided  evidence  that  satisfies  a  large  portion  of  the  criterion.  “Fully  met”  means  the 
program  provided  evidence  that  completely  satisfies  the  criterion. 


Comprehensive.  A  cost  estimate  should  include  costs  of  the  program  over 
its  full  life  cycle,  provide  a  level  of  detail  appropriate  to  ensure  that  cost 
elements  are  neither  omitted  nor  double-counted,  and  document  all  cost- 
influencing  ground  rules  and  assumptions.  The  cost  estimate  should  also 
completely  define  the  program  and  be  technically  reasonable. 

We  found  that  the  cost  estimate  for  GCSS-Army  was  fully 
comprehensive.  The  cost  estimate  included  both  government  and 
contractor  costs  of  the  program  over  its  life  cycle — from  the  inception  of 
the  program  through  design,  development,  deployment,  and  operation 
and  maintenance.  The  cost  estimate  also  included  an  appropriate  level  of 
detail,  which  provided  assurance  that  cost  elements  were  neither  omitted 
nor  double-counted,  and  included  documentation  of  all  cost-influencing 
ground  rules  and  assumptions.  The  cost  estimate  documentation  included 
the  purpose  of  the  cost  estimate,  a  technical  description  of  the  program, 
and  technical  risks  (e.g.,  the  resolution  for  any  identified  deficiencies). 

Well-documented.  A  cost  estimate  should  be  supported  by  detailed 
documentation  that  describes  how  it  was  derived  and  how  the  expected 
funding  will  be  spent  in  order  to  achieve  a  given  objective.  The 
documentation  should  capture  such  things  as  the  source  data  used,  the 
calculations  performed,  the  results  of  the  calculations,  the  estimating 
methodology  used  to  derive  each  work  breakdown  structure  element’s 
cost,  and  evidence  that  the  estimate  was  approved  by  management.  The 
documentation  should  discuss  the  technical  baseline  description,  and  the 
data  in  the  technical  baseline  should  be  consistent  with  the  cost  estimate. 
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We  found  that  the  cost  estimate  for  GCSS-Army  was  substantially  well- 
documented.  The  cost  estimate  captured  such  things  as  the  calculations 
performed  to  derive  each  element’s  cost  and  the  results  of  the 
calculations.  The  documentation  also  included  a  technical  baseline 
description  that  provided  data  consistent  with  the  cost  estimate.  Further, 
the  GCSS-Army  Program  Management  Office  presented  evidence  of 
receiving  approval  of  the  estimate  through  briefings  to  management. 
Although  program  management  officials  did  not  provide  us  with  written 
documentation  of  the  source  data,  the  Office  of  the  Deputy  Assistant 
Secretary  of  the  Army  for  Cost  and  Economics  (DASA-CE)  did  provide  us 
with  a  full  deployment  decision  briefing,25  which  showed  each  major  cost 
element  and  listed  the  methodology  and  sources  of  the  data.  However, 
the  briefing  documents  included  a  limited  amount  of  the  actual  source 
data,  and  we  could  not  determine  their  reliability.26  Without  sufficient 
background  information  about  the  source  data  and  reliability  of  the  data, 
the  GCSS-Army  cost  estimator  cannot  know  with  any  confidence  whether 
the  data  collected  can  be  used  directly  or  need  to  be  modified  before  use 
in  the  cost  estimate. 

Accurate.  A  cost  estimate  should  provide  for  results  that  are  unbiased, 
are  not  overly  conservative  or  optimistic,  and  contain  no  major  mistakes. 
A  cost  estimate  should  be  based  on  an  assessment  of  most  likely  costs 
(adjusted  properly  for  inflation),  updated  to  reflect  significant  changes  and 
grounded  in  a  historical  record  of  cost  estimating  and  actual  experiences 
on  other  comparable  programs.  In  addition,  variances  between  planned 
and  actual  costs  should  be  documented,  explained,  and  reviewed,  and 
estimating  techniques  for  each  cost  element  should  be  used 
appropriately. 

We  found  that  the  cost  estimate  for  GCSS-Army  was  substantially 
accurate.  The  GCSS-Army  cost  model  detailed  the  inflation  indexes  and 
properly  applied  the  indexes  to  each  relevant  cost  element  and  included 
time  phasing  of  the  costs.  The  GCSS-Army  cost  model  did  not  include 


25The  DASA-CE  provides  the  Army  decision  makers  with  cost,  performance  and  economic 
analysis  in  the  form  of  expertise,  models,  data,  estimates  and  analyses  at  all  levels. 

26 GCSS-Army:  Full  Deployment  Decision  Recommended  Army  Cost  Position  Briefing, 
presented  to  Stephen  Barth,  Office  of  the  Deputy  Assistant  Secretary  of  the  Army  for  Cost 
and  Economics  on  October  11,  2012,  and  GCSS-Army:  Army  Enterprise  Systems 
Integration  Program  Independent  Cost  Estimate  Results  Briefing,  presented  to  the  Cost 
Review  Board  Working  Group  in  September  2012. 
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any  major  mistakes,  and  all  its  cost  elements  summed  up  properly  and 
were  consistent  with  the  cost  estimate.  In  addition,  the  estimating 
techniques  (i.e.,  engineering  build-up27)  used  to  create  the  estimate  were 
used  appropriately.  The  cost  model  documentation  did  not  explain 
whether  the  cost  estimate  was  updated  to  reflect  changes  in  technical  or 
program  assumptions.  The  program  management  officials  provided 
documentation  that  reflected  the  technical  changes  for  the  major 
deployment  decisions,  but  the  documentation  did  not  include  details  on 
how  the  costs  were  updated.  Unless  such  documentation  is  available  to 
verify  that  the  cost  estimate  is  properly  updated  on  a  regular  basis, 
management  will  not  have  reasonable  assurance  that  the  cost  estimate 
provides  accurate  information  to  make  informed  decisions  about  the 
program. 

Credible.  A  cost  estimate  should  discuss  any  limitations  of  the  analysis 
because  of  uncertainty  or  biases  surrounding  data  or  assumptions.  The 
cost  estimate  should  include  a  sensitivity  analysis  that  identifies  a  range 
of  possible  costs  based  on  varying  major  assumptions  and  data.  A  risk 
and  uncertainty  analysis  should  be  conducted  to  determine  the  level  of 
risk  associated  with  the  cost  estimate  and  identify  the  effects  of  changing 
key  cost  driver  assumptions  and  factors.  In  addition,  the  estimate’s  results 
should  be  cross-checked  and  reconciled  to  an  independent  cost  estimate 
to  determine  whether  other  estimating  methods  produce  similar  results. 

We  found  that  the  cost  estimate  was  partially  credible.  The  Army  Cost 
Review  Board  developed  an  independent  cost  estimate  that  was 
reconciled  to  the  program  management  officials’  cost  estimate.  The 
program  management  officials’  cost  estimate  mentioned  results  of  a  risk 
analysis;  however  the  risk  and  uncertainty  analysis  was  not  documented. 
Further,  since  the  cost  estimate  that  was  provided  discussed  risk  only  at  a 
summary  level,  it  is  unclear  how  management  considered  risk  related  to 
the  program.  Without  a  fully  documented  risk  and  uncertainty  analysis, 
the  estimate  will  lose  credibility  and  management’s  decision-making 
ability  will  be  impaired  because  it  will  not  know  the  level  of  confidence 
associated  with  achieving  the  cost  estimate.  In  addition,  program 
management  officials  provided  a  cost  estimate  that  identified  major  cost 
drivers,  including  system  deployment  and  training.  The  cost  estimate 


27The  engineering  build-up  method  develops  the  cost  estimate  by  rolling  up  detailed 
estimates  prepared  at  lower  levels  of  the  work  breakdown  structure. 
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documentation  contained  a  reference  that  a  sensitivity  analysis  was 
completed  on  these  cost  drivers,  but  results  of  this  analysis  were  not 
documented.  As  a  result,  the  GCSS-Army  cost  estimator  will  not  have  a 
clear  understanding  of  how  each  major  cost  driver  is  affected  by  a  change 
in  a  single  assumption  and  thus  which  cost  driver  most  affects  the  cost 
estimate.  Further,  GCSS-Army  program  officials  provided  us  with  one 
example  of  evidence  that  indicated  that  some  cross-checking  was 
performed  using  cost  models;  however,  the  results  of  this  cross-checking 
were  not  documented.  The  purpose  of  cross-checking  is  to  determine 
whether  alternative  methods  would  produce  similar  results,  which  would 
increase  the  credibility  of  the  estimate. 

In  our  October  2010  report,  we  recommended  that  the  Army  update  the 
GCSS-Army  cost  estimate  by  using  actual  costs  and  preparing  a 
sensitivity  analysis.28  For  the  2012  cost  estimate,  we  found  that  the  Army 
had  made  progress,  but  we  continued  to  identify  deficiencies  in 
documentation  related  to  the  sensitivity  analysis,  risk  and  uncertainty 
analysis,  and  cross-checking  of  major  cost  elements  for  reasonableness. 


Conclusions 


While  the  Army  made  some  improvements  to  the  schedule  and  cost 
estimates  that  supported  the  full  deployment  decision,  the  Army  did  not 
fully  meet  best  practices  in  developing  schedule  and  cost  estimates  for 
the  GCSS-Army  program.  The  Army  made  progress  in  incorporating 
schedule  best  practices,  such  as  capturing  and  sequencing  all  activities 
and  integrating  activities  horizontally  and  vertically,  but  we  identified  other 
deficiencies  in  schedule  and  cost  best  practices.  For  example,  GCSS- 
Army  did  not  meet  best  practices  related  to  schedule  durations,  a  valid 
critical  path,  and  a  cost  sensitivity  analysis.  It  is  critical  to  correct  the 
deficiencies  identified  with  the  schedule  and  cost  estimates  to  help 
ensure  that  the  projected  spending  for  this  program  is  being  used  in  the 
most  efficient  and  effective  manner.  By  incorporating  best  practices  for 
developing  reliable  schedule  and  cost  estimates,  DOD  would  increase  the 
probability  of  GCSS-Army  successfully  achieving  full  deployment  by  the 
fourth  quarter  of  fiscal  year  2017  to  provide  needed  functionality  for 
financial  improvement  and  audit  readiness. 


28GAO-1 1-53. 
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Recommendations  for  'mProve  implementation  of  GCSS-Army,  we  recommend  that 

the  Secretary  of  the  Army  take  the  following  two  actions: 

Executive  Action 

•  Ensure  that  the  Under  Secretary  of  the  Army,  in  his  capacity  as  the 
Chief  Management  Officer,  directs  the  GCSS-Army  Program 
Management  Office  to  develop  an  updated  schedule  that  fully 
incorporates  best  practices,  including 

•  assigning  resources  to  all  activities, 

•  establishing  durations  of  all  activities, 

•  confirming  that  the  critical  path  is  valid,  and 

•  ensuring  reasonable  total  float. 

•  Ensure  that  the  Under  Secretary  of  the  Army,  in  his  capacity  as  the 
Chief  Management  Officer,  directs  the  GCSS-Army  Program 
Management  Office  to  update  the  cost  estimate  to  fully  incorporate 
best  practices  by  documenting  the  results  of 

•  a  risk  and  uncertainty  analysis, 

•  the  cross-checking  of  major  cost  elements  to  see  if  results  are 
similar,  and 

•  a  sensitivity  analysis. 


Agency  Comments 
and  Our  Evaluation 


We  provided  a  draft  of  this  report  to  DOD  for  review  and  comment.  In  its 
written  comments,  reprinted  in  appendix  III,  DOD  concurred  with  our 
recommendation  to  update  the  schedule  to  fully  incorporate  best 
practices  and  described  planned  and  ongoing  actions  that  the  department 
is  taking  to  address  the  recommendation.  In  particular,  DOD  indicated 
that  the  Army  has  taken  steps  to  help  ensure  that  (1)  all  activities  are 
assigned  resources  in  the  schedule  software,  (2)  all  schedule  activities 
with  long  durations  have  been  detailed,  (3)  level  of  effort  activities  and 
date  constraints  have  been  removed  from  the  schedule  so  that  they  do 
not  define  the  critical  path,  and  (4)  the  majority  of  the  schedule  activities 
associated  with  high  total  float  have  been  removed.  If  effectively 
implemented,  these  actions  should  address  the  intent  of  our 
recommendation. 


DOD  also  concurred  with  our  recommendation  to  update  the  cost 
estimate  to  fully  incorporate  best  practices  by  documenting  the  results  of 
a  risk  and  uncertainty  analysis,  the  cross-checking  of  major  cost  elements 
to  see  if  results  are  similar,  and  a  sensitivity  analysis.  DOD  described 
completed  actions  that  the  department  has  taken  to  address  the 
recommendation.  DOD  stated  that  GCSS-Army  achieved  Milestone  C  in 
August  201 1  and  a  full  deployment  decision  in  December  2012,  and  that 
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it  prepared  a  cost  estimate  per  DOD  acquisition  rules  and  guidelines. 

DOD  also  stated  that  the  Army  (1)  followed  all  Army  directed  best 
practices  and  approvals  from  the  Office  of  the  Deputy  Assistant  Secretary 
of  the  Army  for  Cost  and  Economics  and  (2)  prepared  a  sensitivity 
analysis,  a  risk  analysis,  and  cross-checked  major  cost  elements  for 
similar  results,  but  that  those  documented  analyses  and  results  were  not 
included  in  the  formal  cost  estimates  as  directed  by  the  Army.  DOD 
commented  that  these  documented  analyses  and  results  are  part  of  the 
formal  working  papers  and  were  provided  to  GAO  in  February  2013. 

However,  these  actions  do  not  fully  address  the  intent  of  our 
recommendation.  As  stated  in  our  report,  we  focused  on  the  extent  to 
which  GCSS-Army’s  schedule  and  cost  estimates  were  prepared 
consistent  with  GAO’s  Schedule  and  Cost  Guides.  We  reviewed  the  cost 
estimate  documentation  provided  by  the  Army  in  February  2013  and 
additional  information  provided  in  February  2014  and  determined  that  the 
documentation  did  not  fully  meet  best  practices  for  a  risk  and  uncertainty 
analysis,  a  sensitivity  analysis,  and  cross-checking  of  major  cost 
elements  for  similar  results.  As  stated  in  our  report,  GCSS-Army  program 
management  officials  provided  a  cost  estimate  that  mentioned  the  results 
of  a  risk  and  uncertainty  analysis,  and  contained  a  reference  that  a 
sensitivity  analysis  was  completed.  Also,  GCSS-Army  program 
management  officials  provided  us  with  one  example  of  evidence  that 
indicated  some  cross-checking  was  performed  using  cost  models. 
However,  the  results  of  the  risk  and  uncertainty  and  sensitivity  analyses, 
as  well  as  the  cross-checking  were  not  documented  consistent  with  best 
practices.  As  stated  in  our  report,  incorporating  best  practices  for  a 
reliable  cost  estimate  would  help  ensure  that  DOD  has  a  reliable  cost 
estimate  that  provides  the  basis  for  effective  resource  allocation, 
proactive  course  correction  when  warranted,  and  accountability  for 
results. 


As  agreed  with  your  offices,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  until  30  days  from  the 
report  date.  At  that  time,  we  will  send  copies  to  the  appropriate 
congressional  committees;  the  Secretary  of  Defense;  the  Secretary  of  the 
Army;  the  Assistant  Secretary  of  Defense  (Acquisition);  the  Acting  Deputy 
Chief  Management  Officer;  the  Under  Secretary  of  Defense  (Comptroller); 
the  Under  Secretary  of  the  Army,  in  his  capacity  as  the  Chief 
Management  Officer  of  the  Army;  and  the  Program  Manager  for  GCSS- 
Army.  In  addition,  the  report  will  be  available  at  no  charge  on  the  GAO 
website  at  http://www.gao.gov. 
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If  you  or  your  staffs  have  any  questions  about  this  report,  please  contact 
Asif  A.  Khan  at  (202)  512-9869  or  khana@gao.gov  or  Nabajyoti  Barkakati 
at  (202)  512-4499  or  barkakatin@gao.gov.  Contact  points  for  our  Offices 
of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last 
page  of  this  report.  GAO  staff  members  who  made  key  contributions  to 
this  report  are  listed  in  appendix  IV. 


Asif  A.  Khan 
Director 

Financial  Management  and  Assurance 


Nabajyoti  Barkakati 
Chief  Technologist 
Applied  Research  and  Methods 
Center  for  Science,  Technology,  and  Engineering 
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Appendix  I:  Assessment  of  the  Global 
Combat  Support  System-Army  Program’s 
Schedule  Estimate 


This  appendix  provides  the  results  of  our  analysis  of  the  extent  to  which 
the  Global  Combat  Support  System-Army  schedule  supporting  the 
December  2012  full  deployment  decision  met  the  characteristics  of  a 
high-quality,  reliable  schedule.  Table  3  provides  the  detailed  results  of  our 
analysis. 

GAO’s  methodology  includes  five  levels  of  compliance  with  its  best 
practices.1  “Not  met”  means  the  program  provided  no  evidence  that 
satisfies  any  of  the  criterion.  “Minimally  met”  means  the  program  provided 
evidence  that  satisfies  a  small  portion  of  the  criterion.  “Partially  met” 
means  the  program  provided  evidence  that  satisfies  about  half  of  the 
criterion.  “Substantially  met”  means  the  program  provided  evidence  that 
satisfies  a  large  portion  of  the  criterion.  “Fully  met”  means  the  program 
provided  evidence  that  completely  satisfies  the  criterion. 


^AO,  GAO  Schedule  Assessment  Guide:  Best  Practices  for  Project  Schedules — 
Exposure  Draft,  GAO-12-120G  (Washington,  D.C.:  May  2012). 
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Appendix  I:  Assessment  of  the  Global  Combat 
Support  System-Army  Program’s  Schedule 
Estimate 


Table  3:  Assessment  of  Global  Combat  Support  System-Army  (GCSS-Army)  Schedule  Estimate  Compared  to  Best  Practices 


Characteristic  Best  practice 


Assessment  Rationale  for  assessment 


Comprehensive  The  schedule  should  reflect  all  Partially  met 
activities  as  defined  in  the 
project’s  work  breakdown 
structure,  which  defines  in  detail 
the  work  necessary  to  accomplish 
a  project’s  objectives,  including 
activities  to  be  performed  by  both 
the  government  and  contractors. 

The  schedule  should  reflect  the 
resources  (e.g.,  labor,  materials, 
and  overhead)  needed  to  do  the 
work,  whether  those  resources  will 
be  available  when  needed,  and 
whether  any  funding  or  time 
constraints  exist.  The  schedule 
should  realistically  reflect  how 
long  each  activity  will  take, 
allowing  for  discrete  progress 
measurement  with  specific  start 
and  finish  dates. 


The  schedule  reflected  all  activities  to  be  performed  by  both 
the  government  and  the  contractor  for  the  program.  To  be 
comprehensive,  the  program  schedule  should  reflect  all 
efforts  necessary  to  successfully  complete  the  program. 

However,  no  resources  were  loaded  in  the  schedule 
software  or  assigned  to  specific  activities  in  the  schedule. 
Program  management  officials  told  us  that  the  contractor 
managed  resources  by  using  a  separate  system  outside  the 
schedule.  The  amount  of  available  resources  affects 
estimates  of  work  and  its  duration  and  resources  that  will  be 
available  for  subsequent  activities.  Information  on  resource 
needs  and  availability  in  each  work  period  assists  the 
program  office  in  forecasting  the  likelihood  that  activities  will 
be  completed  as  scheduled.  Because  the  current  schedule 
did  not  allow  insight  into  the  current  or  projected  allocation  of 
resources,  the  risk  of  program  slippage  could  be  significantly 
increased. 

Several  of  the  activity  durations  in  the  schedule  did  not  meet 
the  standard  best  practice  for  activity  duration  (i.e. ,  shorter 
than  approximately  44  working  days  or  2  working  months). 
Specifically,  about  30  percent  of  the  remaining  activities 
were  greater  than  44  working  days.  When  durations  are  not 
based  on  the  effort  required  to  complete  an  activity,  the 
resources  available,  resource  efficiency,  and  other  factors 
such  as  previous  experience  on  similar  activities,  then  there 
is  little  confidence  in  meeting  the  target  deliverable  date. 
Because  the  estimates  of  work  required  and  duration  for  an 
activity  are  tied  to  the  availability  of  resources,  the  lack  of 
such  information  could  hinder  management’s  ability  to 
compute  total  labor  and  equipment  hours,  calculate  total 
project  and  per-period  cost,  resolve  resource  conflicts,  and 
establish  the  reasonableness  of  the  plan. 
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Appendix  I:  Assessment  of  the  Global  Combat 
Support  System-Army  Program’s  Schedule 
Estimate 


Characteristic 

Best  practice 

Assessment 

Rationale  for  assessment 

Well-constructed  The  schedule  should  be  planned  Partially  met 
so  that  critical  project  dates  can 
be  met.  All  activities  should  be 
logically  sequenced — that  is,  listed 
in  the  order  in  which  they  are  to 
be  carried  out — with  predecessor 
and  successor  logic.  Date 
constraints  and  lags  should  be 
minimized  and  justified  to  help 
ensure  that  the  interdependence 
of  activities  that  collectively  lead  to 
the  completion  of  events  or 
milestones  can  be  established 
and  used  to  guide  work  and 
measure  progress.  The  schedule 
should  identify  the  program  critical 
path — the  path  of  longest  duration 
through  the  sequence  of 
activities — to  determine  the 
program’s  earliest  completion 
date.  The  schedule  should  identify 
total  float — the  amount  of  time  by 
which  a  predecessor  activity  can 
slip  before  the  delay  affects  the 
program’s  estimated  finish  date — 
so  that  the  schedule’s  flexibility 
can  be  accurately  determined.  As 
a  general  rule,  activities  along  the 
critical  path  have  the  least  amount 
of  float. 


The  majority  of  logic  used  to  sequence  the  activities  within 
the  schedule  was  generally  error  free,  clearly  indicating  the 
order  of  activities  that  must  be  accomplished.  The  majority  of 
the  activities  in  the  schedule  exhibited  a  straightforward 
sequencing  of  logic — finish-to-start  logic  relationship. 

Our  analysis  found  that  the  GCSS-Army  schedule’s  critical 
path  was  not  valid  because  it  included  level  of  effort  activities 
and  date  constraints.  Level  of  effort  activities,  such  as 
program  management,  should  not  define  the  critical  path 
because  they  are  nondiscrete  support  activities  that  do  not 
produce  a  definite  end  product;  therefore,  level  of  effort 
activities  cannot  determine  the  length  of  the  project.  In 
addition,  date  constraints  prevent  the  critical  path  from  being 
a  continuous  sequence  of  events  from  the  current  to  finish 
dates.  Rather  than  relying  on  constraints,  the  schedule 
should  use  logic  and  durations  in  order  to  reflect  realistic 
start  and  completion  dates  for  activities.  Until  the  schedule 
can  produce  a  true  critical  path,  the  program  office  will  not 
be  able  to  provide  reliable  timeline  estimates  or  identify 
when  problems  or  changes  may  occur  and  their  effect  on 
downstream  work.  Also,  without  a  valid  critical  path, 
management  cannot  focus  on  activities  that  will  have 
detrimental  effects  on  the  key  project  milestones  and 
deliveries  if  they  slip. 

Our  analysis  found  that  float  calculations  within  the  schedule 
were  not  reliable  because  the  schedule  was  being  updated 
to  reflect  a  modification  to  the  system.  The  schedule  could 
not  be  updated  until  the  modification  was  formally  approved, 
and  this  caused  the  negative  float  values  in  the  schedule. 
Twenty-eight  percent  of  the  remaining  schedule  activities 
had  total  float  greater  than  1 00  working  days,  meaning  they 
could  slip  almost  5  working  months  and  not  affect  the 
estimated  finish  date  of  the  program.  In  addition,  11  activities 
had  negative  float,  ranging  from  -1  to  -10  days.  Negative 
float  stems  from  constraining  one  or  more  activities  or 
milestones  in  the  network.  Negative  float  indicates  critical 
path  effort  that  may  require  management  action,  such  as 
overtime  or  resequencing  of  work.  The  constraint  should  be 
examined  and  justified,  and  the  resulting  negative  float 
should  be  evaluated  for  reasonableness.  Without  reasonable 
float  estimates,  management  may  be  unable  to  identify 
activities  that  could  be  permitted  to  slip  and  thus  release  and 
reallocate  resources  to  activities  that  require  more  resources 
to  be  completed  on  time. 
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Appendix  I:  Assessment  of  the  Global  Combat 
Support  System-Army  Program’s  Schedule 
Estimate 


Characteristic 

Best  practice 

Assessment 

Rationale  for  assessment 

Credible  The  schedule  should  be  Substantially 

horizontally  integrated,  meaning  met 
that  it  should  link  products  and 
outcomes  associated  with  other 
sequenced  activities.  These  links 
are  commonly  referred  to  as 
“handoffs”  and  serve  to  verify  that 
activities  are  arranged  in  the  right 
order  to  achieve  aggregated 
products  or  outcomes.  The 
schedule  should  also  be  vertically 
integrated,  meaning  that  the  dates 
for  starting  and  completing 
activities  in  the  integrated  master 
schedule  should  be  aligned  with 
the  dates  for  supporting  tasks  and 
subtasks.  Such  mapping  or 
alignment  among  levels  enables 
different  groups  to  work  to  the 
same  master  schedule. 

A  schedule  risk  analysis  should  be 
performed  using  statistical 
techniques  to  predict  the  level  of 
confidence  in  meeting  a  program’s 
completion  date,  determine  the 
time  contingency  needed  for  a 
level  of  confidence,  and  identify 
high-priority  risks  and 
opportunities.  The  analysis  should 
focus  not  only  on  critical  path 
activities  but  also  on  activities 
near  the  critical  path,  since  they 
can  affect  the  program’s  status. 


The  schedule  for  GCSS-Army  appropriately  exhibited 
horizontal  and  vertical  integration. 

However,  the  program  office  had  not  fully  conducted  a 
schedule  risk  analysis.  Although  the  program  office  provided 
documentation  for  a  schedule  risk  analysis  performed  by  the 
contractor,  the  documentation  did  not  show  that  a  schedule 
risk  analysis  was  performed  for  all  supporting  activities.  Risk 
assessments  were  only  provided  for  5  of  the  1 1  supporting 
activities  in  the  schedule.  In  addition,  there  was  no  mention 
of  correlation  between  activity  durations,  and  no  data  were 
provided  that  would  show  which  schedule  activities  were 
given  high,  medium,  or  low  risk  factors.  If  a  schedule  risk 
analysis  is  not  conducted,  (1)  program  management  cannot 
determine  the  likelihood  of  meeting  the  program’s 
completion  date,  (2)  how  much  schedule  risk  contingency  is 
needed  to  provide  an  acceptable  level  of  certainty  for 
completion  by  a  specific  date,  (3)  risks  most  likely  to  delay 
the  program,  (4)  how  much  contingency  reserve  each  risk 
requires,  and  (5)  the  paths  or  activities  that  are  most  likely  to 
delay  the  program.  Unless  a  statistical  simulation  is  run, 
calculating  the  completion  date  from  schedule  logic  and  the 
most  likely  duration  distributions  will  tend  to  underestimate 
the  program’s  overall  critical  path  duration. 


Page  23 


GAO-14-470  DOD  Business  Systems  Modernization 


Appendix  I:  Assessment  of  the  Global  Combat 
Support  System-Army  Program’s  Schedule 
Estimate 


Characteristic 

Best  practice 

Assessment 

Rationale  for  assessment 

Controlled  The  schedule  should  be  Substantially 

continually  updated  using  logic  met 
and  durations  to  realistically 
forecast  dates  for  program 
activities.  A  schedule  narrative 
should  accompany  the  updated 
schedule  to  provide  decision 
makers  and  auditors  a  log  of 
changes  and  their  effect,  if  any,  on 
the  schedule  time  frame. 

The  schedule  should  be  analyzed 
continually  for  variances  to 
determine  when  forecasted 
completion  dates  differ  from 
planned  dates.  This  analysis  is 
especially  important  for  those 
variations  that  impact  activities 
identified  as  being  on  a  program’s 
critical  path  and  that  can  impact  a 
scheduled  completion  date.  A 
baseline  schedule  is  the  basis  for 
managing  the  program  scope,  the 
time  period  for  accomplishing  it, 
and  the  required  resources.  The 
baseline  schedule  is  designated 
the  target  schedule  and  is  subject 
to  a  configuration  management 
control  process  against  which 
program  performance  can  be 
measured,  monitored,  and 
reported.  A  corresponding 
baseline  document  explains  the 
overall  approach  to  the  program; 
defines  custom  fields  in  the 
schedule  file;  details  ground  rules 
and  assumptions  used  in 
developing  the  schedule;  and 
justifies  constraints,  lags,  long 
activity  durations,  and  any  other 
unique  features  of  the  schedule. 


GCSS-Army  program  management  officials  stated  that  they 
met  weekly  to  discuss  proposed  schedule  changes  and 
update  the  schedule’s  progress,  and  management  officials 
prepared  a  schedule  narrative  document  that  contained  a  list 
of  custom  fields  and  assumptions.  In  addition,  we  found  no 
anomalies  throughout  the  schedule  (e.g.,  activities  with 
planned  start  dates  scheduled  to  occur  in  the  past  and 
activities  with  actual  finish  dates  scheduled  to  occur  in  the 
future). 

Although  there  was  not  a  documented  original  baseline 
schedule  against  which  program  performance  could  be 
measured,  program  officials  updated  the  schedule  status 
weekly,  performed  trend  analysis  and  met  regularly  to  track 
progress.  Without  a  formally  established  baseline  schedule 
to  measure  performance  against,  management  cannot 
identify  or  mitigate  the  effect  of  unfavorable  performance.  A 
well-documented  schedule  is  essential  for  validating  and 
defending  a  baseline  schedule,  analyzing  changes  in  the 
program  schedule,  and  identifying  the  reasons  for  variances 
between  estimates. 


Source:  GAO  analysis  based  on  information  provided  by  the  Army.  |  GAO-14-470 
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This  appendix  provides  the  results  of  our  analysis  of  the  extent  to  which 
the  Global  Combat  Support  System-Army  cost  estimate  supporting  the 
December  2012  full  deployment  decision  met  the  characteristics  of  a 
high-quality  cost  estimate.  Table  4  provides  the  detailed  results  of  our 
analysis. 

GAO’s  methodology  includes  five  levels  of  compliance  with  its  best 
practices.1  “Not  met”  means  the  program  provided  no  evidence  that 
satisfies  any  of  the  criterion.  “Minimally  met”  means  the  program  provided 
evidence  that  satisfies  a  small  portion  of  the  criterion.  “Partially  met” 
means  the  program  provided  evidence  that  satisfies  about  half  of  the 
criterion.  “Substantially  met”  means  the  program  provided  evidence  that 
satisfies  a  large  portion  of  the  criterion.  “Fully  met”  means  the  program 
provided  evidence  that  completely  satisfies  the  criterion. 


Table  4:  Assessment  of  Global  Combat  Support  System-Army  (GCSS-Army)  Cost  Estimate  Compared  to  Best  Practices 


Characteristic  Best  practice 


Assessment 


Rationale  for  assessment 


Comprehensive  The  cost  estimate  includes  all  Fully  met 
life-cycle  costs.  The  cost 
estimate  completely  defines  the 
program,  reflects  the  current 
schedule,  and  is  technically 
reasonable.  The  cost  estimate 
work  breakdown  structure  is 
product  oriented,  traceable  to 
the  statement  of  work,  and  at  an 
appropriate  level  of  detail  to 
ensure  that  cost  elements  are 
neither  omitted  nor  double- 
counted.  The  estimate 
documents  all  cost-influencing 
ground  rules  and  assumptions. 


The  GCSS-Army  cost  estimate  included  both  government 
and  contractor  costs  of  the  program  over  its  life  cycle.  The 
documentation  provided  a  description  of  the  purpose  of  the 
estimate,  a  technical  description  of  the  program  and 
technical  risks  (e.g.,  the  resolution  for  any  identified 
deficiencies). 

The  work  breakdown  structure  outlined  the  end  product  and 
major  work  to  be  performed.  The  program  provided 
supporting  documentation  that  showed  the  ground  rules  and 
assumptions,  such  as  hardware  requirements.  The  estimate 
was  based  on  a  standard  cost  element  structure  as  stated  in 
the  Department  of  Defense  Automated  Information  Systems 
Economic  Analysis  Guide. 


^AO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 
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Characteristic  Best  practice 


Assessment  Rationale  for  assessment 


Well-documented  The  documentation  should  Substantially 

capture  the  source  data  used, 
the  reliability  of  the  data,  and 
how  the  data  were  normalized. 

The  documentation  describes  in 
sufficient  detail  the  calculations 
performed  and  the  estimating 
methodology  used  to  derive 
each  work  breakdown  structure 
element’s  cost.  The 
documentation  describes  step- 
by-step  how  the  estimate  was 
developed  so  that  a  cost  analyst 
unfamiliar  with  the  program 
could  understand  what  was 
done  and  replicate  it.  The 
documentation  discusses  the 
technical  baseline  description 
and  the  data  in  the  baseline  are 
consistent  with  the  estimate. 

The  documentation  provides 
evidence  that  the  cost  estimate 
was  reviewed  and  accepted  by 
management. 


met  The  documentation  for  the  cost  estimate  captured  such 
things  as  the  calculations  performed  to  derive  each 
element’s  cost  and  the  results  of  the  calculations.  The 
documentation  also  included  a  technical  baseline 
description  that  provided  data  consistent  with  the  cost 
estimate.  The  GCSS-Army  Program  Management  Office 
presented  evidence  of  receiving  approval  of  the  estimate 
through  briefings  to  management. 

However,  documentation  to  capture  the  source  data  used 
and  the  reliability  of  the  data  for  the  cost  estimate  was 
limited.  The  program  management  officials  said  that  the 
source  data  used  for  the  cost  estimate  are  not  documented 
within  the  GCSS-Army  cost  model  because  of  size 
limitations  of  the  software,  but  that  the  source  data  are 
stored  by  the  Office  of  the  Deputy  Assistant  Secretary  of  the 
Army  for  Cost  and  Economics  (DASA-CE).  The  DASA-CE 
provided  us  with  a  full  deployment  decision  briefing  given  to 
Army’s  Cost  Review  Board  Working  Group,  which  showed 
each  major  cost  element  and  listed  the  methodology  and 
sources  of  the  data.  However,  this  briefing  document 
displayed  a  limited  amount  of  the  source  data,  and  we  could 
not  determine  the  reliability  of  the  source  data.  Without 
sufficient  background  information  about  the  source  data  and 
their  reliability,  the  GCSS-Army  cost  estimator  cannot  know 
with  any  confidence  whether  the  data  collected  can  be  used 
directly  or  need  to  be  modified  before  use  in  the  cost 
estimate. 


Page  26 


GAO-14-470  DOD  Business  Systems  Modernization 


Appendix  II:  Assessment  of  the  Global  Combat 
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Estimate 


Characteristic  Best  practice 


Assessment  Rationale  for  assessment 


Accurate  The  cost  estimate  results  are  Substantially 

unbiased,  not  overly 
conservative  or  optimistic  and 
based  on  an  assessment  of 
most  likely  costs.  The  estimate 
has  been  adjusted  properly  for 
inflation.  The  estimate  contains 
few,  if  any,  minor  mistakes.  The 
cost  estimate  is  regularly 
updated  to  reflect  significant 
changes  in  the  program  so  that 
it  is  always  reflecting  current 
status.  The  estimate  is  based  on 
a  historical  record  of  cost 
estimating  and  actual 
experiences  from  other 
comparable  programs. 

Variances  between  planned  and 
actual  costs  are  documented, 
explained,  and  reviewed. 


met  The  GCSS-Army  cost  model  detailed  the  inflation  indexes 
supporting  the  estimated  costs  and  did  not  include  any 
major  mistakes.  All  its  cost  elements  summed  up  properly 
and  were  consistent  with  the  cost  estimate.  In  addition,  the 
estimating  techniques  used  to  create  the  cost  estimate  were 
used  appropriately.  However,  the  amount  of  risk  and 
contingency  was  not  documented,  so  it  is  unclear  whether 
the  cost  estimate  is  unbiased,  not  overly  conservative  or 
optimistic,  or  based  on  the  assessment  of  most  likely  costs. 
Unless  the  cost  estimate  is  based  on  an  assessment  of  the 
most  likely  costs  and  reflects  the  degree  of  uncertainty  given 
all  of  the  risks  considered,  management  will  not  be  able  to 
make  good  program  decisions. 

The  cost  estimate  documentation  did  not  explain  whether 
the  cost  estimate  was  updated  to  reflect  changes  in 
technical  or  program  assumptions.  The  program 
management  officials  provided  documentation  that  reflected 
the  technical  changes  for  the  major  deployment  decisions, 
but  the  documentation  did  not  include  details  on  how  the 
costs  were  updated.  Unless  properly  updated  on  a  regular 
basis,  the  cost  estimate  cannot  provide  management  with 
accurate  information  for  assessing  alternative  decisions. 
Certain  elements  in  the  cost  estimate  appeared  to  be 
current  and  based  on  actual  historical  information,  such  as 
hourly  rates  and  costs  per  license;  but  there  is  a  lack  of 
documentation  regarding  the  source  of  the  data  that  does 
not  allow  us  to  determine  the  reliability  of  the  data  or  if  they 
were  appropriate  for  this  estimate. 

The  cost  estimate  does  not  include  any  discussion  of 
variances  between  planned  and  actual  costs,  which  would 
enable  GCSS-Army  cost  estimators  to  assess  how  well  they 
are  estimating  program  costs  and  identifying  lessons 
learned.  However,  program  officials  told  us  that  they  tracked 
the  differences  between  the  funds  planned  and  expended 
for  the  major  deployment  decisions.  Without  a  documented 
comparison  between  the  current  estimate,  which  is  updated 
with  actual  cost,  and  the  old  estimate,  management  cannot 
see  how  well  it  is  estimating  program  costs  and  how  the 
program  is  changing  overtime. 
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Characteristic 

Best  practice 

Assessment 

Rationale  for  assessment 

Credible  A  risk  and  uncertainty  analysis  Partially  met 

was  conducted  that  quantified 
the  imperfectly  understood  risks 
and  identified  the  effects  of 
changing  key  cost  driver 
assumptions  and  factors.  Major 
cost  elements  were  cross¬ 
checked  to  see  whether  results 
were  similar.  An  independent 
cost  estimate  was  conducted  by 
a  group  outside  the  acquiring 
organization  to  determine 
whether  other  estimating 
methods  produce  similar  results. 

The  cost  estimate  includes  a 
sensitivity  analysis  that  identifies 
a  range  of  possible  costs  based 
on  varying  major  assumptions, 
parameters,  and  data  inputs. 


The  Army  Cost  Review  Board  developed  an  independent 
cost  estimate  that  was  reconciled  to  the  program 
management  officials’  cost  estimate. 

Program  management  officials  provided  a  cost  model  that 
mentioned  results  of  a  risk  and  uncertainty  analysis,  but  the 
results  of  this  analysis  were  not  documented.  Since  the  cost 
estimate  discussed  risk  only  at  a  summary  level,  it  is  unclear 
how  management  considered  risk  related  to  the  program. 
Without  a  fully  documented  risk  and  uncertainty  analysis, 
the  estimate  will  lose  credibility  and  management’s  decision¬ 
making  ability  will  be  impaired  because  it  will  not  know  the 
level  of  confidence  associated  with  achieving  the  cost 
estimate. 

GCSS-Army  program  management  officials  provided  us  with 
one  example  of  evidence  that  indicated  some  cross¬ 
checking  was  performed  using  cost  models;  however,  the 
results  of  this  cross-checking  were  not  documented.  The 
purpose  of  cross-checking  is  to  determine  whether 
alternative  methods  would  produce  similar  results,  which 
would  increase  the  cost  estimate’s  credibility. 

Program  management  officials  provided  a  cost  estimate  that 
identified  major  cost  drivers  (e.g.,  system  deployment  and 
training).  The  cost  estimate  documentation  contained  a 
reference  that  a  sensitivity  analysis  was  completed  on  these 
cost  drivers,  but  results  of  this  analysis  were  not 
documented.  As  a  result,  the  GCSS-Army  cost  estimator  will 
not  have  a  clear  understanding  of  how  each  major  cost 
driver  is  affected  by  a  change  in  a  single  assumption  and 
thus  which  cost  driver  most  affects  the  cost  estimate. 


Source:  GAO  analysis  based  on  information  provided  by  the  Army.  |  GAO-14-470 
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ACQUISITION 


ASSISTANT  SECRETARY  OF  DEFENSE 

30 1 5  DEFENSE  PENTAGON 
WASHINGTON,  DC  2030 1-3015 


SEP  17  2014 


Mr.  Asif  A.  Khan. 

Director,  Financial  Management  and  Assurance 
and  Applied  Research  and  Methods 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 


Dear  Mr.  Khan: 


This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report,  GAO- 14- 
470,  “DOD  BUSINESS  SYSTEMS  MODERNIZATION:  Additional  Enhancements  Are  Needed 
for  Army  Business  System  Schedule  and  Cost  Estimates  to  Fully  Meet  Best  Practices,”  dated 
August  6,  2014  (GAO  Code  1971 12).  Detailed  comments  on  the  report  recommendations  are 
enclosed. 


Sincerely, 


Katrina  McFarland 
Assistant  Secretary  of  Defense 
(Acquisition) 


Enclosure: 
As  stated 
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GAO  Draft  Report  Dated  August  6, 2014 
GAO-14-470  (GAO  CODE  197112) 

“DOD  BUSINESS  SYSTMES  MODERNIZATION:  ADDITIONAL  ENHANCEMENTS 
ARE  NEEDED  FOR  ARMY  BUSINESS  SYSTEM  SCHEDULE  AND  COST  ESTIMATES 
TO  FULLY  MEET  BEST  PRACTICES” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  DRAFT  REPORT 


RECOMMENDATION  1:  To  help  improve  the  implementation  of  GCSS-Army,  the 
Government  Accountability  Office  (GAO)  recommends  that  the  Secretary  of  the  Army  take  the 
following  two  actions: 

Ensure  that  the  Under  Secretary  of  the  Amiy,  in  his  capacity  as  the  Chief  Management  Officer, 
directs  the  GCSS-Army  Program  Management  Office  to  develop  an  updated  schedule  that  fully 
incorporates  best  practices,  including 

•  assigning  resources  to  all  activities, 

•  establishing  durations  of  all  activities, 

•  confirming  that  the  critical  path  is  valid,  and 

•  ensuring  reasonable  total  float. 

DoD  RESPONSE:  Concur. 

Assigning  resources  to  all  activities:  GCSS-Army  has  worked  through  the  LSI  and  DCMA  to 
make  sure  all  tasks  are  resource  loaded  in  the  MPM  tool  the  LSI  uses. 

Establishing  durations  of  all  activities:  Early  on  in  the  project  some  tasks  with  long  durations 
may  have  made  it  through  the  rolling  wave  planning  window.  GCSS-Army  has  since  worked 
on  making  sure  all  scheduled  work  in  the  planning  window  is  detailed  out. 

Confirming  that  the  critical  path  is  valid:  There  have  been  several  steps  we  have  taken  to  make 
sure  this  has  occurred.  With  permission  from  DCMA  we  have  removed  LOE  out  of  the  IMS. 
Therefore,  the  critical  path  will  only  go  through  measureable  work  activities. 

Second,  the  program  exercises  options  on  the  contract  one  at  a  time.  In  the  current  option,  the 
Government  PMO  knows  the  milestones  that  need  to  occur  on  time  as  to  not  affect  the  next 
option.  Those  are  the  release  milestones.  The  LSI  and  Government  Program  Management 
Office  met  and  agreed  GCSS-Army  would  have  critical  paths  leading  to  each  release  in  the 
current  option.  When  the  next  option  is  exercised,  we  will  be  able  to  link  to  these  milestones 
and  see  new  critical  paths  through  to  the  next  option. 

GCSS-Army  has  also  removed  the  date  constraints  throughout  the  IMS. 
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Ensuring  reasonable  total  float:  The  LSI  has  removed  license  renewals  and  maintenance 
upgrades  from  the  IMS.  These  tasks  were  able  to  slip  to  the  end  of  the  option  period 
because  of  the  type  of  work,  without  affecting  the  critical  path.  These  tasks  represented  a 
majority  of  the  tasks  with  high  total  float. 

GCSS-Army  still  has  tasks  in  IMS  that  will  lead  to  future  work  in  options  yet  to  be  exercised. 
Those  tasks  still  have  high  floats.  However,  when  these  tasks  are  late  to  their  baseline,  the 
LSI  reports  them  weekly  to  the  Government  PMO  so  that  actions  can  be  taken  in  case  the 
Government  PMO  foresees  the  tasks  affecting  the  exercising  of  future  options. 

RECOMMENDATION  2:  To  help  improve  the  implementation  of  GCSS-Army,  the 
Government  Accountability  Office  (GAO)  recommends  that  the  Secretary  of  the  Army  take  the 
following  two  actions: 

Ensure  that  the  Under  Secretary  of  the  Army,  in  his  capacity  as  the  Chief  Management  Officer, 
directs  the  GCSS-Army  Program  Management  Office  to  update  the  cost  estimate  to  fully 
incorporate  best  practices,  by  documenting  the  results  of 

•  a  risk  and  uncertainty  analysis, 

•  the  cross  checking  of  major  cost  elements  to  see  if  results  are  similar,  and 

•  a  sensitivity  analysis. 

DoD  RESPONSE:  Concur  and  complete.  As  part  of  the  GCSS-Army  achieving  a  Milestone  C 
in  August  2011  and  a  Full  Deployment  Decision  in  December  2012,  it  prepared  a  cost  estimate 
per  DoD  acquisition  rules  and  guidelines.  GCSS-Army  followed  all  Anny  directed  best 
practices  and  approvals  from  the  Office  of  the  Deputy  Assistant  Secretary  of  the  Army  for  Cost 
and  Economics  (ODASA-CE).  GCSS-Army  did  prepare  a  sensitivity  analysis,  risk  analysis  and 
cross-checked  major  cost  elements  for  similar  results,  however  those  documented  analyses  and 
results  were  not  included  in  the  fomtal  cost  estimates  as  directed  by  the  Anny.  These 
documented  analyses  and  results  are  part  of  the  formal  working  papers  of  the  program  and 
ODASA-CE,  and  were  provided  to  GAO  in  February  2013  during  the  audit  process. 


Page  31 


GAO-14-470  DOD  Business  Systems  Modernization 


Appendix  IV:  GAO  Contacts  and  Staff 
Acknowledgments 


GAO  Contacts 

Asif  A.  Khan,  (202)512-9869  orkhana@gao.gov 

Nabajyoti  Barkakati,  (202)  512-4499  or  barkakatin@gao.gov 

Staff 

Acknowledgments 

In  addition  to  the  contacts  named  above,  Arkelga  Braxton  (Assistant 
Director),  Karen  Richey  (Assistant  Director),  Beatrice  Alff,  Tisha 

Derricotte,  Jennifer  Echard,  Emile  Ettedgui,  Patrick  Frey,  and  Jason  Lee 
made  key  contributions  to  this  report. 

(197112) 

Page  32  GAO-14-470  DOD  Business  Systems  Modernization 

GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  website  (http://www.gao.gov).  Each  weekday 
afternoon,  GAO  posts  on  its  website  newly  released  reports,  testimony, 
and  correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted 
products,  go  to  http://www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  website, 

http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 
Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 
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Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512- 
4400,  U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room 
7125,  Washington,  DC  20548 
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Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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